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(54) System for providing document change information for a community of users 



(57) A document recommendation system incorpo- 
rating a document change monitoring agent (403). For a 
document recommendation system to be effective, it is 
desirable to enable users to be cognizant of changes 
that may occur to the document. The present invention 
addresses this issue by coupling a document change 
monitoring agent (403), which automatically detects 
changes in referenced documents, with a recommender 
system (401), which helps users share and evaluate 
information in a collaborative way. One advantage of the 
invention is that it brings human judgement into the rel- 
evance evaluation of the detected changes and allows 
the results to be shared with other people likely to be 
interested, in such a way that redundant work is 
decreased. Another advantage of the invention is that it 
enables Systems Administrators of a document recom- 
mendation system to be more efficient in the manage- 
ment of the system. 
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Description 

[0001] The present invention is related to the field of information retrieval, and in particular to the field of providing 
document change information to a user. 
5 [0002] The ever-increasing universe of electronic information, for example as found on the World Wide Web (herein 
after referred to as the Web), competes for the effectively fixed and limited attention of people. Both consumers and pro- 
ducers of information want to understand what kinds of information are available, how desirable it is, and how its con- 
tentand use change through time. 

[0003] Making sense of very large collections of linked documents and foraging for information in such environ- 
w ments is difficult without specialized aids. Collections of linked documents are often connected together using hypertext 
links. The basic structure of linked hypertext is designed to promote the process of browsing from one document to 
another along hypertext links, which is unfortunately very slow and inefficient when hypertext collections become very 
large and heterogeneous. Two sorts of aids have evolved in such situations. The first are structures or tools that abstract 
and cluster information in some form of classification system. Examples of such would be library card catalogs and the 
15 Yahoo! Web site (URL http://www.yahoo.com). The second are systems that attempt to predict the information relevant 
to a user's needs and to order the presentation of information accordingly. Examples would include search engines 
such as Lycos (URL: http://www.lycos.com), which take a user's specifications of an information need, in the form of 
words and phrases, and return ranked lists of documents that are predicted to be relevant to the user's need. 
[0004] Another class of tools is recommender systems. Recommender systems provide a list of recommended 
20 subsequent web pages worth viewing based on some predetermined filtering criteria. One such recommender tool is 
the "Recommend'' feature provided on the Alexa Internet Web site (URL: http://www.alexa.com). The "Recommend" 
feature provides a list of related Web pages that a user may want to retrieve and view based on the Web page that they 
are currently viewing. 

[0005] Another recommender system is termed "the Knowledge Pump" and is described by Glance, N., Arregui, D. 
25 and Dardenne, M. "Knowledge Pump: Supporting the Flow and Use of Knowledge." In Information Technology for 
Knowledge Management. Eds. U. Borghoff and R. Pareschi, New York: Springer-Verlag, pp. 35-45, 1998. The "Knowl- 
edge Pump" was designed for use within organizations and has a key focus on the sharing of information in the form of 
documents. 

[0006] Prospective applications of recommender systems are as information sharing applications for organizations. 
30 Here, the quality of the recommendation service is key. If the recommender system fails to recognize stale references 
and changes in recommended documents, the users' experience of the information sharing environment will suffer sig- 
nificantly as a result. In other words, a main failing of recommender systems currently is that they provide static recom- 
mendations of potentially dynamic objects. 

[0007] Changes to documents are also of interest to users. Such changes can take many forms: substantive con- 
35 tent change, cosmetic/syntactic changes, and disappearance of the document In addition, administrators of data 
stores containing document references face the additional problem that the same document can be referenced by mul- 
tiple references. 

[0008] Some repositories and databases are equipped with tools, using triggers, that help users deal with these 
problems by notifying users when referenced items of interest change in some way or are removed. On the WWW, there 
40 are now also several services available to help users monitor Web pages based on their Uniform Resource Locator 
(URL) address, such as: 

netmind: http://www.netmind.com . 

45 Smart Bookmarks: http://www.fii^oor.corn/SmartBookmarks2.0/QuickSta rthtml . 

Grassroots: http://www-pcd.stanford.edu/Grassroots. 

"WebTracker - a Web Service for tracking documents, 11 by Fishkin, K. and Bier, E. available at 
so httpy/www.pare.xerox.com/istl/members/fishkin/doc/ webtracker.html . 

[0009] These services notify users when URLs they have registered with the service have changed in same way. 
In addition, they may also be able to federate the change monitoring systems of other kinds of networked repositories. 
However, evaluating the nature of the change and its importance for the user is a difficult task to automate and is thus 
55 the weakness of such change monitoring systems. The agent notifications while perhaps including a great deal of data 
concerning the change, may be potentially irrelevant to the user, and in the long run, the high noise vs. signal ratio may 
cause the user more annoyance than aid. The most accurate way to evaluate the nature and importance the change 
remains the user him/herself, at least for the foreseeable future. 
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[0010] In addition, within a given workgroup, work community, or organization, it is likely that the existence of sub- 
stantive changes in a document will be relevant to a number of people, not just one. In this case, the work of evaluating 
the nature of the change is likely to be done not once but many times, as there is currently no good way to share this 
work. 

5 [0011] A document recommendation system incorporating a document change monitoring agent is disclosed. As 
described above with respect to the prior art, there are a number of problems that are experienced by users of net- 
worked repositories. The present invention addresses several that stem from the fact that access to these repositories 
is typically done via document references that point to documents. The problems that arise are: (1) the document ref- 
erence itself may not remain stable (may become stale); (2) the content of document referenced may change; and (3) 

w multiple references may exist for the same item. Solutions that exist today, such as document change monitoring agents 
on the World Wide Web and change trigger agents within repositories and databases, are entirely automated proc- 
esses, and as such, do not resolve these problems satisfactorily. In addition, they are oriented to single users as 
opposed to groups of users. 

[0012] The present invention addresses these issues by coupling a document change monitoring agent, which 
15 automatically detects changes in referenced documents, with a recommender system, which helps users share and 
evaluate information in a collaborative way. An important advantage of the present invention is that it brings human 
judgement into the relevance evaluation of the detected changes and allows the results to be shared with other people 
likely to be interested, in such a way that redundant work is decreased. 

[0013] The present invention shows how this limitation can be overcome by, first of all, coupling a document change 
20 monitoring agent with the recommender system and, secondly, implementing a collaborative process for evaluating the 
nature and importance of these changes. It has been determined through research and experience with existing rec- 
ommender systems, that such functionality is key to improving the usability of a recommender system intended for shar- 
ing information in the form of documents. 

[0014] A system implementation of the present invention may include the following elements: a document recom- 
25 mendation element for providing document recommendations to a user based on a user profile, said document recom- 
mendation element further comprising one or more action elements for responding to a change in a recommended 
document, each of said one or more action elements performing a different action based on the nature of said change; 
document representation storage, for storing representations of said recommended documents; a document registra- 
tion element coupled to said document recommendation element, said document registration element for enabling a 
30 user to identify recommended documents for which they would like to be notified of changes; a document change mon- 
itoring element coupled to said document registration element, said document change monitoring element for detecting 
a change to said document and reporting said change to said document recommendation element; and a user notifica- 
tion element coupled to said document recommendation element, said user notification element for notifying users of 
said change. 

35 

Figure 1 is a block diagram of a change monitoring agent as is currently known in the prior art; 
Figure 2 a block diagram of a recommender system as is currently known in the prior art; 
Figure 3 is a flowchart illustrating the steps for coupling a change monitoring agent with a recommender system as 
may be performed in the currently preferred embodiment of the present invention; 
40 Figure 4 is a block diagram further illustrating the interactions between the change monitoring agent, the recom- 
mender system, the users and the document repositories as may be performed in the currently preferred embodi- 
ment of the present invention; and, 

Figure 5 is a block diagram of a computer system which may be utilized to implement the system of the currently 
preferred embodiment of the present invention. 

45 

[0015] A system for providing document change information to a community of users having an interest in the doc- 
ument is disclosed. The system provides for collaborative mechanism for user to share feedback about such changes. 
The currently preferred embodiment of the present invention is implemented for analyzing collections of linked docu- 
ments (i.e. Web Pages) residing on the portion of the Internet known as the World Wide Web (hereinafter the Web). The 

so Web is the portion of the Internet that is used to store and access linked documents. A Web page may have multi-media 
content as well as relative and absolute links to other pages. A Web page is accessed by its Uniform Address Locator 
(URL) address. A Web Site is a collection of related Web pages associated with an entity having a presence on the 
World Wide Web such as a company, educational institute or the like. A repository (or source/document repository) is 
a Web location containing collections of documents which may be accessed via the Web. 

55 [0016] A Web Browser is a tool which enables a user to traverse through and view documents residing on the Web. 
Other rendering means associated with the Browser will permit listening to audio portions of a document or viewing 
video or image portions of a document. Examples of commercially available Web Browsers include Netscape Naviga- 
tor®, available from Netscape Communications Corporation, and Internet Explorer, available from the Microsoft® Cor- 
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p oration. 

[0017] In the present invention a document change monitoring agent is coupled with a recommender system for 
three main purposes: (1) to more effectively evaluate the nature and importance of changes in referenced documents; 
(2) to share such results among a community of users; and (3) to help administrators better manage data stores con- 

5 taining document references (e.g., the data store associated with a recommender system). 

[0018] The document change monitoring agent, as described in detail below, is responsible for tracking changes in 
referenced documents. Its main components, with respect to this invention, are data storage for storing representations 
of the document content and a function for evaluating the extent and character of the change. Such document change 
monitoring agents exist already on the Web (as described in the prior art) and have also been instantiated as triggers 

10 within databases. Added to the document change monitoring agent is means to share information with the second mod- 
ule, the recommender system. 

[001 9] A recommender system is a system that enables users to share recommendations in a way that limits infor- 
mation overload. An important component of the recommender system is a filtering mechanism that, for each user, 
ranks or rates the priority of recommended items coming from others. In the present invention, we are concerned only 

15 with recommender systems that handle recommendations of pointers to items (as opposed to the items themselves). 
[0020] By coupling the document change monitoring agent with a recommender system, users who have recom- 
mended items which have since changed can be notified of that change, if they so desire. The recommender system is 
extended to provide users with the means to evaluate the nature and importance of the change. This evaluation can 
then be shared with other users of the recommender system, using the filtering mechanisms that already exist. 

20 [0021] TTius, the present invention, on the one hand, augments the automatic process of monitoring document 
changes with human evaluation, and on the other hand, limits the amount of redundant evaluation work that goes on by 
sharing the results of the evaluation with those other people most likely to be interested. 

[0022] Finally, a number of side benefits ensue from the perspective of administrators of the recommender system 
(or other data store connected to the combined system): (1) the administrator is notified when pointers become stale; 

25 (2) the administrator is notified when two or more references point to the same document; (3) the administrator is noti- 
fied when the reference itself has changed; and (3) the administrator is notified when it is likely that access to a partic- 
ular repository is no longer possible, i.e., when that repository has been disconnected from the network. 
[0023] Described in more detail below are both document change monitoring agents and recommender systems 
as they exist today, and additionally a characterization of the nature of the source repositories alongside which these 

30 operate. Then discussed in greater depth is the invention and manner for coupling of the change monitoring agent and 
the recommender system in order to resolve the problems discussed above. Finally, discussed are the changes 
required in both the document change monitoring agent and in a recommender system in order to implement the 
present along with explicit specifications for both. 

35 Document Change Monitoring Agent 

[0024] Figure 1 illustrates a document change monitoring agent as known in the prior art. A document change mon- 
itoring agent provides a means to track changes in online documents and to notify interested people about the change. 
Document monitoring involves three steps which are illustrated with reference to Figure 1. 

40 

Step 1 User Document Registration and Interest Specification : The user 101 registers with the monitoring agent 
1 02 the identifier of the online document 1 03 (for agents operating on the Web, this is typically the URL of the doc- 
ument). Additionally some agents permit the definition of: i) a region to monitor in the document, ii) a list of key- 
words that represent the user interest, iii) the periodicity of the check the agent will perform. 

45 

Step 2 Monitoring Agent Monitors Document(s) : The monitoring agent 102 monitors document changes by period- 
ically accessing the document or, alternatively, by requiring a notification service from the document repository. The 
latter case is less common, since i) document repositories often do not offer this service, ii) the added value of the 
agent resides then mainly in the federation of several repositories for monitoring changes. 

so In the former case, the monitoring agent 1 02 is responsible for detecting the change and therefore associates with 
the document identifier a record of information representing the state of the document. Such information is stored 
in document content representation storage 1 04. The monitoring agent 1 02 detects changes by comparing succes- 
sive records. The information recorded may range from the whole document content to a document content check- 
sum of only a few bytes. The relevance and efficiency of the change detection derives directly from the sort of 

55 record(s) (or combination thereof) that is used. 

Step 3 Monitoring Agent Notification To User The monitoring agent 102 notifies the user 101 about the change, 
typically by sending an email describing the detected change. Here again, the description of the change can range 
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from a detailed textual comparison to an almost empty message notifying the occurrence of the change, depending 
on which kind of records were kept for the document. 

Recomm nder Systems 

5 

[0025] Figure 2 is a block diagram illustrating a recommender system as is known in the prior art. A recommender 
system uses filtering techniques to provide personalized recommendations of documents to individual users. Referring 
to Figure 2, the minimal set of features common to any recommender system is: 
[0026] Data storage 201 , which supports at least the following three data structures: 

10 

• User: personal and profiling information for a member of the system. 

• Document: reference to a document (typically a URL). 

• Review: associates a user, a document and a numerical score (rating). 

is [0027] A filtering mechanism 202 which ranks or rates recommended items for any given user, based on the user's 
profile (which can contain the user's history of interaction with the system, keywords). Common filtering approaches 
include automated collaborative filtering and personalized content filtering, which are described in greater detail in the 
following references: Resnick et al. "GroupLens: An Open Architecture for Collaborative Filtering of Netnews". Proceed- 
ings of the ACM 1994 Conference on Computer Supported Cooperative Work, Chapel Hill, NC, 199', pp. 175-186; and 

20 Shardanand et al. Social Information Filtering: Algorithms for Automating Word of Mouth. Proceedings of the ACM 1995 
Conference on Computer Human Interactions, Denver, CO, 1995, pp. 210-217. 

[0028] A User interface 203 providing an interface for displaying recommended items to a user based on predicted 
preferences; an interface for the user to provide feedback on recommended items; an interface for allowing the user to 
recommend new items. It should be noted that design of such user interfaces are well known in the art, and thus could 
25 be implemented using such known technology. Thus, no further discussion on the creation of the user interfaces is 
deemed necessary. 

Source repositories 

30 [0029] Source repositories are defined here as any repository that can serve as a source of recommendations (of 
documents). One principle example is the Web, in which every document is referenced by a URL. Other examples 
include networked databases and document management systems. Accessing a source repository via a reference and 
the appropriate communication channel can result in three different results: 

35 1 . The document is accessed without problems; 

2. The reference is no longer valid (for example, a URL that points to a page that no longer exists); 

3. The source repository is not responding (for example, the HTTP server upon which the source repository resides 
and may be down or taken off the network). 

40 Coupling A Document Change Monitoring Agent With A Recommender System 

[0030] A method to couple a document change monitoring agent and recommender systems that solves the prob- 
lems described above is described with reference to the flowchart of Figure 3. First, the recommender system registers 
the recommended document to the monitoring agent, step 301. This will in turn will monitor changes occurring to the 
45 document. 

[0031 ] The monitoring agent reports to the recommender system any change (change in content or location as well 
as disappearance of document) occurring to the document, step 302. The monitoring agent also detects multiple reg- 
istered documents having the same content and reports about these duplicates. 

[0032] Responsive to these reports, the recommender system reacts and takes action to the reports as followed: 

50 

Step 303, for relocated documents, it stores the new location of the document, registers that new location to the 
agent and un registers the previous one. 

- Step 304, for documents whose content has changed, it notifies users implicated by the change (i.e. users having 
55 shown interest for the recommended document and the document submitter, if any). These users are given the 
possibility to annotate again the document or to comment on the change. Each user's evaluation of the change is 
made available to the others, as well as to the membership of the recommender system when they subsequently 
receive/access/review the recommendation. 
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Step 305, for unavailable documents, it shows on the display that the document is no longer available and possibly 
notifies the system administrator or the user who recommended the document 

Step 306, for duplicate references to the same document, the recommender system updates its representation of 
5 the document to track the multiple references. 

[0033] Based on user requests responsive to their review of community feedback relating to the change, the rec- 
ommender system might unregister a document from the change monitoring agent, step 307. 
[0034] Figure 4 further illustrates the interactions between the agent, the recommender system, the users and the 

10 document repositories. Referring to Figure 4, the recommender system 401 will provide recommended documents to 
users 402. To perform this function, the recommender system 401 interacts with the document repositories 404. The 
document repositories 404 will typically reside on the Web. In any event, the users 402 may then elect to be informed 
about changes to the recommended documents. When this occurs, recommender system 401 registers the recom- 
mended documents with monitoring agent 403. Upon detecting a document change, the monitoring agent 403 reports 

is back to the recommender system 401 information about the change. The recommender system 401 then notifies the 
users 402 of the change. The user may then provide feedback related to the change. 
[0035] The method of the present invention solves the problems cited above in the following ways: 

• Dynamic documents (i.e. subject to changes) are handled gracefully; users are given the possibility to review col- 
20 laboratively the detected changes and to modify their recommendation accordingly. 

• Importance of changes is collaboratively evaluated in the frame of the recommender system. 

• Multiple persons no longer need to same effort/work to assess the value of the change, and in particular, insignifi- 
cant changes need only be identified once. 

• Broken links (open or heterogeneous systems) are detected by the agent, reported to the recommender system, 
25 which in turn is given the possibility to react appropriately. 

• Multiple entries having the same content are detected by the agent, reported to the recommender system, which 
in turn is given the possibility to react appropriately. 

[0036] Provided herein is an example, a specification of a monitoring agent (which does not include either the man- 

30 agement of broken links, of relocated documents, or of multiple entries). 

[0037] The agent as described below runs independently from the recommender system. At regular interval it 
invokes its own DoMonitoring() method. It should provide some data storage capabilities (however this job could be 
taken over by the recommender system) and should export and Application Programming Interface (API). The basic 
persistent data structure at this level is the Monitor. One can understand coupling as accessing and manipulating mon- 

35 itors through the API both by the monitoring agent and the recommender system. 

Monitor 
[0038] 

40 

Document 

Reference to a document's content 
45 LastModified 

Last date and time the document has been found modified. Initially Unknown. 
Status 

50 

Ok if the document's content has been accessed without problems, Error otherwise. Initially Ok. 
Characteristic 

55 Value computed using the document's content, see the private methods of the agent provided below. Initially 

its value equals to None. 

Subscribers 
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List of users interested in being aware of the changes for this particular document. Initially Unknown. 
API methods : 
5 For managing the monitors : 
[0039] 

Monitor (Document, User) 

10 

Creates a new monitor for a document with a first subscriber. 
Get Monitors() 
is Returns all stored monitors. 

GetMonitor (Document) 

Returns the monitor for a document or None if there is not such a monitor. 

20 

AddMonitor (Monitor) 
Adds a monitor. 
25 RemoveMonitor (Monitor) 
Removes a monitor. 
UpdateMonitor (Monitor) 

30 

Updates the attributes of an existing monitor. 
For executing the monitoring process : 
35 [0040] 

DoMonitoringO : 

mons = GetMonitors () 
40 for m in mons: 

d = m. Document 
(sc, c) = FetchContent (d) 

45 
50 
55 
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if (sc = Ok) : 

pc = Parse (c) 
tn. Status = Ok 

if m. Characteristic = None: 

nw Characteristic » Characterize (pc) 

else: 

old = m. Characteristic 
new » Characterize (pc) 
if (not Compare (old, new)): 
m.LastModified = Now() 
m. Characteristic = new 

else : 

m.LastModified = Now() 
m. Status = Error 
UpdateMonitor (m 

Private Methods For Monitor Agent : 
[0041] 

FetchContent (Document) 

Retrieves a documents content (using its URL for example) from a source repository, returns a tuple (Status- 
Code, Content). The result is reported via a StatusCode returned by the FetchDocument private method of the 
monitoring agent, which can have three possible values: 

• Ok: Document accessed without problems. 

• Deprecated Reference: Reference is no longer valid (for example URL pointing to a page that doesn't exist any 
longer). 

• Access Failure: Major problem, the source repository is not responding (an HTTP server may be down, for 
example). 

NowO 

Returns the current date and time. 
Parse (content) 

Parses the content. For example, removing all punctuation and HTML mark-ups. Returns parsed-content. 
Characterize (parsed-content) 

Returns a value that will remain the same if the document's content remains unchanged. For example, the 
returned value might be generated by a hashcode function. 

Compare (Characteristic, Characteristic) 
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Compares two characteristics produced by the Characterize method and returns True if they are considered to 
be similar enough, Fals oth rwis . A threshold value will probably be involved in the implementation. 

Recommender System: Minimal Additional Requirements 

[0042] The recommender system of the currently preferred embodiment builds upon existing systems. Such sys- 
tems would provide the capabilities described above (e.g. a filtering mechanism and various user interfaces for enabling 
the user to interact with the system). To couple the recommender system with the specified monitoring agent requires 
at least: 

(1) A way for the recommender system to subscribe references to be monitored to the monitoring agent; see the 
Subscribe {Document d, User u) method below. 

(2) A way to periodically ask for, or to be notified of, the list of changes the monitoring agent has detected; see the 
CheckDocuments() method below. 

(3) A way to allow users of the recommender system to turn monitoring on and off for any particular reference; e.g., 
a unsubscribe (Document d t Useru) method. 

(4) A way to notify the user that the content has changed (after suitable filtering) and propose one of several 
actions, e.g., evaluate (review) the change, ignore, stop monitoring, advise deletion of reference from data store of 
recommender system; see the Notifyllser {Monitor m, Users) method below. 

(5) A way to notify the administrator when the reference is unavailable over a long period of time (one week, say); 
see the NotifyAdmin {Monitor m) below. 

[0043] Specifically, the recommender system will at regular intervals invokes its own CheckDocumentsO method. 
The time of last invocation is stored in the LastCheck attribute and a reference to the monitoring agent is kept in the 
MonitoringAgent attribute. 

[0044] In the reviewing interface one additional option is provided, Subscribe, which turns monitoring on (or off) for 
the particular reference. 
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Subscribe (Document d, User u) : 

m = Moni toringAgen t . GetMoni t or ( d) 
if (m = None) : 

m = new Monitor (d, u) 

Moni toringAgen t.UpdateMonitor (m) 

Else: 

Subscribers . add ( u) 
Moni toringAgen t .UpdateMoni tor (m) 
Moni toringAgen t.AddMoni tor (m) 
CheckDocuments () : 

/nons = Moni toringAgen t. GetMonitors () 
for in in jnons; 

if (jn.LastModif ied > LastCheck) : 
if m. Status = Ok: 

for s in ;n. Subscribers : 
NotifyUser (m, s) 
else : 

Not if yAdmin (zn) 
LastCheck = Now() 
NotifyUser [Moni tor m, User s) : 

[0045] Notifies the users that the content of the reference m. Document has changed and proposes several options: 
Review 

Fill review about changes that will be accessible to all the other users. 
Ignore 

Do nothing, discard the notification. 
Unsubscribe 

m. Subscribers. remove (s) 
if (jn. Subscribers « [] ) : 

Moni toringAgent. RemoveMonitor (in) 

else 

Moni toringAgent . UpdateMoni tor (m) 
Not if yAdmin {Monitor m) : 
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[0046] Notifies the administrator of the recommender system that the document corresponding to the reference 
m.Document is not accessible. 

[0047] It should be noted that the monitoring agent and recommender system may execute on either the same or 
different computer systems. Further, a since communication is preferably through a predetermined Application Pro- 
5 gramming Interface, multiple such monitor agents may operate with a single recommender system, or multiple recom- 
mender system may operate with a single monitor agent. Implementation of such a configuration would not depart from 
the spirit and scope of the present invention. 

Overview of a Computer Controlled Display System In the Currently Preferred Embodiment of the Present 
10 Invention 

[0048] The computer based system on which the currently preferred embodiment of the present invention may be 
implemented is described with reference to Figure 5. As noted above, the present invention is implemented using soft- 
ware programming instructions for execution on a computer based system. The computer based system and associ- 

15 ated operating instructions (e.g. software) embody circuitry used to implement the present invention. Referring to Figure 
5, the computer based system is comprised of a plurality of components coupled via a bus 501. The bus 501 may con- 
sist of a plurality of parallel buses (e.g. address, data and status buses) as well as a hierarchy of buses (e.g. a processor 
bus, a local bus and an I/O bus). In any event, the computer system is further comprised of a processor 502 for execut- 
ing instructions provided via bus 501 from Internal memory 503 (note that the Internal memory 503 is typically a com- 

20 bination of Random Access and Read Only Memories). The processor 502 will be used to perform various operations 
in support extracting raw data from Web site, converting the raw data into the desired feature vectors and topology, 
usage path and text similarity matrices, categorization and spreading activation. Instructions for performing such oper- 
ations are retrieved from Internal memory 503. Such operations that would be performed by the processor 502 would 
include the processing steps described in the flowcharts of Figure 3 and the accompanying descriptions. The opera- 

25 tions would typically be provided in the form of coded instructions in a suitable programming language using well-known 
programming techniques. The processor 502 and Internal memory 503 may be discrete components or a single inte- 
grated device such as an Application Specification Integrated Circuit (ASIC) chip. 

[0049] Also coupled to the bus 501 are a keyboard 504 for entering alphanumeric input, external storage 505 for 
storing data, a cursor control device 506 for manipulating a cursor, a display 507 for displaying visual output and a net- 
so work connection 508. The keyboard 504 would typically be a standard QWERTY keyboard but may also be telephone 
like keypad. The external storage 505 may be fixed or removable magnetic or optical disk drive. The cursor control 
device 506, e.g. a mouse or trackball, will typically have a button or switch associated with it to which the performance 
of certain functions can be programmed. The network connection 508 provides means for attaching to a network, e.g. 
a Local Area Network (LAN) card or modem card with appropriate software. The network ultimately attached to is the 
35 Internet, but it may be through proxy servers or intermediary networks or dial-up services such as America On-Line, 
Prodigy ® or CompuServe®. 

[0050] Thus, a system for providing document change information for a community of users is disclosed. While the 
present invention is described with respect to a preferred embodiment, it would be apparent to one skilled in the art to 
practice the present invention with other configurations of digital document management systems. Such alternate 
40 embodiments would not cause departure from the spirit and scope of the present invention. For example, the currently 
preferred embodiment discloses two seemingly discrete systems having an application programming interface for sys- 
tem communication, whereas the present invention could be implemented as a single system with the document 
change agent being a part of the recommender system. 

45 Claims 

1 . A system for providing document change information to a user comprising: 

a document recommendation element for providing document recommendations to a user based on a user 
so profile, said document recommendation element further comprising one or more action elements for respond- 

ing to a change in a recommended document, each of said one or more action elements performing a different 
action based on the nature of said change; 

document representation storage, for storing representations of said recommended documents; 
a document registration element coupled to said document recommendation element, said document registra- 
55 tion element for enabling a user to identify recommended documents for which they would like to be notified of 

changes; 

a document change monitoring element coupled to said document registration element, said document 
change monitoring element for detecting a change to said document and reporting said change to said docu- 
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ment recommendation element; and 

a user notification element coupled to said document recommendation element, said user notification element 
for notifying users of said change. 

s 2. The system as recited in claim 1 , wherein one of said one or more action elements takes action responsive to one 
or more of the following: 

a change in recommended document location, and wherein said action is to modify location information for 
said recommended document in said document representation storage; 
10 a change in the content of said recommended document, and wherein said action taken is to notify users of 

said recommended document of said change, to solicit feedback of said change, and to provide said feedback 
to other users of said recommended document; 

a determination that a recommended document has become unavailable, and wherein said action taken is to 
notify a systems administrator or the user who recommended the document; and 
15 a determination that there exist duplicate references to said recommended document, and wherein said action 

taken is to modify location information for said recommended document in said document representation stor- 
age to reflect the multiple references. 

3. The system as recited in claim 1 or claim 2, wherein said user is a systems administrator. 

20 

4. A method for providing document change information responsive to instances of change of said documents, in a 
system for managing the recommendation of documents to a user, said documents contained in a collection of net- 
work accessible documents, said method comprising the steps of: 

25 (a) a user indicating to said system that they would like to be notified of a change to a recommended document; 

(b) said system registering said recommended document with a monitoring agent, said monitoring agent for 
detecting changes to network accessible documents; 

(c) said monitoring agent indicating to said system that a change to said recommended document has 
occurred; 

30 (d) said system taking an appropriate action according to a type of change to said recommended document. 

5. A method as recited in claim 4, wherein said type of change to said recommended document is one or more of: 

a change in the content of said recommended document, and wherein said action taken is to notify users of 
35 said recommended document of said change, to solicit feedback of said change, and to provide said feedback 

to other users of said recommended document; 

a determination that a recommended document has become unavailable; and wherein said action taken is to 
notify a systems administrator or the user who recommended the document; and 

a determination that there exist duplicate references to said recommended document, and wherein said action 
40 taken is to modify location information for said recommended document in said document representation stor- 

age to reflect the multiple references. 

6. The method as recited in claim 4 or claim 5, wherein said user is a systems administrator. 

45 7. A method for obtaining evaluations of changes in resources accessible to users of a system, the system including: 
resource access circuitry for accessing a set of resources; 

user notification data indicating, for each of a subset of the resources, one or more users to be notified when 
a change in the resource is detected; and 
so evaluation data indicating user evaluations of resources; 

the method comprising: 

(a) using the resource access circuitry to automatically attempt to access one of the resources in the sub- 
set and to automatically obtain access result data indicating the result; 
55 (b) automatically applying a change notification criterion to the access result data to obtain notification 

data indicating whether to notify users that the resource is changed; 

(c) if the notification data indicate that users should be notified that the resource is changed, automatically 
notifying users indicated for the resource by the user notification data that the resource is changed; 
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(d) receiving evaluations of the resource as changed from the notified users; and 

(e) in response to the received evaluations, automatically updating the evaluation data to indicate user 
evaluations of the resource as changed. 

8. The method of claim 7, in which the access result data indicate one or more of the following: 

whether the resource's location is changed and in which the change notification criterion does not require noti- 
fication if the only change in the resource is a change in location; 

whether the resource's content is changed and in which the change notification criterion requires notification if 
the resource's content is changed; and 

whether the resource is available and in which the change notification criterion does not require notification if 
the only change in the resource is a change in availability. 

9. The method of claim 7 or claim 8, in which the resource access circuitry includes a network and the resource is a 
network-accessible document. 

10. The method of any of claims 7 to 9, in which the network is the Internet and step (a) comprises: 

providing a universal resource location (URL) for the resource over the network. 
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